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Investigating  Inter-Organizational  Collaboration  during  the  Haiti  Relief 
Effort  from  a  Macrocognition  Perspective 

Abstract 

On  January  12,  2010,  a  massive  earthquake  ravished  Haiti’s  capital  city  of  Port- 
au-Prince.  Relief  organizations,  governments,  and  people  from  all  over  the  world 
poured  into  Haiti  to  help  address  the  devastating  circumstances.  Throughout  the 
relief  effort  the  communications  that  transpired  between  responders  were 
recorded  in  the  All  Partners  Area  Network  (APAN)  collaboration  system.  APAN 
data  from  the  Haiti  relief  effort  was  analyzed  for  this  research  where  we 
employed  an  empirical  process  to  evaluate  a  model  of  team  collaboration.  We 
analyzed  the  inter-agency  collaborative  communication  by  applying  definitions  of 
the  macrocognitive  processes  included  in  a  model  of  team  collaboration  to  each 
thought  unit  posted  by  responders.  Macrocognition  is  a  nascent  area  of  knowledge 
engineering  that  focuses  on  understanding  how  cognition  emerges  in  natural 
environments.  The  goal  for  the  research  reported  here  is  to  understand  the  role  of 
cognition  in  teams  who  are  collaborating  to  solve  unique,  challenging, 
ambiguous,  infonnation-rich  problems.  Results  indicate  the  task  environment  will 
influence  which  macrocognitive  processes  are  used  and  we  found  evidence  for 
several  new  macrocognitive  processes.  A  more  complete  model  of  team 
collaboration  can  guide  designers  of  collaboration  tools  to  facilitate  decision 
making  as  part  of  the  overall  task. 


INTRODUCTION 

On  January  12,  2010,  Haiti  experienced  a  7.0  magnitude  earthquake  10  miles  from  its  capital  city 
of  Port-au-Prince.  The  global  relief  effort  that  followed  was  on  a  scale  that  to  date  has  been 
unmatched.  Relief  organizations,  responders  from  many  governments,  and  people  from  all  over 
the  world  poured  into  Haiti  to  help  address  the  devastating  circumstances.  Throughout  the  relief 
effort  the  communications  that  transpired  between  air,  ground,  and  ocean-based  assistance  crews 
were  recorded  in  the  All  Partners  Area  Network  (APAN)  collaboration  system.  APAN  data  from 
the  Haiti  relief  effort  provided  the  data  that  was  analyzed  for  this  research.  The  goal  for  the 
research  reported  here  is  to  understand  the  role  of  cognition  in  distributed  teams  who  are 
collaborating  to  solve  challenging,  unique,  dynamic,  information-rich  problems  and  to  apply  this 
understanding  to  make  recommendations  for  collaboration  support. 

The  research  described  in  this  paper  focuses  on  the  contextually  bound  processes  entailed  in 
sensemaking,  managing  uncertainty,  and  related  cognitive  processes  entailed  in  responding  to 
emerging  events  that  occur  in  dynamic  decision-making  situations.  For  the  research  reported 
here  we  have  examined  team  collaboration  in  the  context  of  the  Haiti  Humanitarian  Assis¬ 
tance/Disaster  Relief  (HA/DR)  operation.  We  elected  to  analyze  this  data  set  as  it  represents  a 
large  set  of  real-world  data  with  all  the  characteristics  of  interest  for  conducting  an  empirical 
evaluation  of  the  model  of  team  collaboration.  That  is,  a  problem  where  ad  hoc  teams  are 
quickly  assembled  to  deal  with  an  emerging  event  that  requires  a  collaborative  effort  to  deal  with 
the  problem-solving  event. 
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Macrocognition  is  an  emerging  field  within  the  area  of  cognitive  engineering  that  describes  the 
way  cognition  occurs  in  naturalistic,  or  real-world,  decision-making  events  (Cacciabue  & 
Hollnagell,  1995).  In  this  view  of  macrocognition,  the  focus  is  on  cognitive  task  analyses  of 
functions  required  to  perform  a  task  or  achieve  a  goal  (Klein,  Ross,  Moon,  Klein,  Hoffman,  & 
Hollnagel,  2003).  Macrocognition  in  teams  (Letsky  et  ah,  2008)  further  expands  the  concept  by 
considering  group  cognition  and  the  collective  (team)  cognitive  processes  that  enable  the 
externalization  of  internalized  knowledge  building.  These  cognitive  functions  are  generally 
perfonned  during  collaborative  team  problem  solving,  where  the  emphasis  is  on  building  new 
knowledge.  Macrocognition  is  differentiated  from  microcognition  in  several  ways. 

Microcognition  places  an  emphasis  on  experimental  control  of  tasks  and  theoretical  accounts  of 
specific  phenomena  while  macrocognition  emphasizes  cognition  and  performance  under  actual 
working  conditions.  Macrocognitive  phenomena  generally  occur  over  longer  time  periods,  have 
ill-defined  goals,  and  do  not  focus  on  the  “basic”  cognitive  functions  of  microcognition  (e.g., 
perception,  attention  and  memory).  Macrocognition  encompasses  cognitive  processes  involved 
in  detecting  problems,  developing  and  sharing  situation  awareness,  generating  options,  using 
analogues,  mentally  simulating  courses  of  action,  planning  and  re-planning,  maintaining 
vigilance,  and  assessing  risk  (Klein  et  ah,  2003). 

The  framework  of  collaborative  problem  solving  developed  as  part  of  the  Office  of  Naval 
Research  (ONR)  Collaboration  and  Knowledge  Interoperability  (CKI)  Program  (Letsky,  Warner, 
Fiore,  &  Salas,  2007;  Warner,  Letsky,  &  Cowen,  2005)  provides  the  conceptual  foundation  for 
this  research.  The  emphasis  on  macrocognition  in  teams  was  initiated  as  part  of  a  larger  issue  of 
how  to  understand  and  facilitate  complex,  collaborative  activity  -  specifically  in  quick-response 
ad  hoc  teams.  Both  commercial  and  military  communities  are  evolving  in  tenns  of  the  socio- 
technical  systems  employed,  globalization,  and  ubiquitous  information  accessibility,  which 
combined  are  changing  the  dynamics  of  team  activity  (Letsky  &  Warner,  2008).  The  CKI 
program  seeks  to  develop  a  better  understanding  of  internalized,  non-quantifiable,  mental 
processes  at  work  as  teams  collect,  filter,  process  and  share  infonnation  for  problem-solving 
purposes. 

The  objective  of  the  CKI  program  is  to  respond  to  emerging  needs  in  both  the  military  and 
business  environments  to  better  understand  and  improve  the  effectiveness  of  team  decision 
making  in  complex,  data-rich  situations.  The  long-range  program  objective  is  to  develop  a  range 
of  cognitive  science  based  tools,  models,  computational  methods,  and  human-agent  interfaces  to 
help  attain  common  situation  awareness  among  distributed  team  members,  engaged  in 
asynchronous,  quick-response  collaboration  for  issue  resolution,  course  of  action  selection,  or 
decision  making.  Our  objectives  for  the  research  reported  here  are  to:  (1)  empirically  evaluate 
the  CKI  model  of  team  collaboration  based  upon  analysis  of  real-world  complex  decision¬ 
making  events,  (2)  determine  which  macrocognitive  processes  are  used  and,  if  necessary,  refine 
the  model  based  on  empirical  analysis,  and  (3)  develop  a  better  understanding  of  the  cognitive 
processes  employed  when  teams  collaborate  to  solve  problems. 

The  remainder  of  this  paper  is  organized  as  follows:  the  background  provides  a  brief  review  of 
the  literature  on  collaboration  and  a  model  of  team  collaboration,  and  describes  the  lexical  link 
analysis  technique  for  identifying  themes  in  a  large  data  set;  the  method  section  describes  the 
data  analysis  and  coding  plan;  the  results  section  discusses  the  qualitative  and  quantitative 
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outcomes  from  our  analysis;  and  the  discussion  section  summarizes  the  results  and  conclusions 
then  outlines  avenues  for  future  research. 


BACKGROUND 

Collaboration  occurs  “when  a  group  of  autonomous  stakeholders  of  a  problem  domain  engage  in 
an  interactive  process,  using  shared  rules,  norms,  and  structures,  to  act  or  decide  on  issues  related 
to  that  domain”  (Wood  &  Gray,  1991,  p.l  1).  This  interactive  process  is  performed  in  a  collabor¬ 
ative  team  environment,  with  collaborative  defined  as  the  “cognitive  aspects  of  joint  analysis  or 
problem  solving  for  the  purpose  of  attaining  shared  understanding  sufficient  to  achieve  situation¬ 
al  awareness  for  decision  making  or  creation  of  a  product”  (Letsky  &  Warner,  2008,  p.  4). 
Benefits  afforded  by  inter-organizational  collaboration  include  better  decision-making  as  a  result 
of  shared  information,  enhanced  coordination  among  dispersed  units,  innovation  resulting  from 
the  cross-pollination  of  ideas,  and  cost  savings  produced  by  sharing  resources  and  the  transfer  of 
smart  practices  (Hansen  &  Nobia,  2004;  Mankin  &  Fitzgerald,  2004).  Team  members  can  often 
provide  several  perspectives  on  an  issue  for  generating,  choosing,  and  implementing  action 
plans. 

The  research  reported  on  in  this  paper  builds  on  a  stream  of  research  where  we  analyzed  and 
coded  transcripts  or  chat  logs  that  transpired  during  five  real-world  problem-solving  events  and 
one  laboratory  experiment  to  empirically  evaluate  a  model  of  team  collaboration  developed 
under  the  CKI  Program.  A  model  of  team  collaboration  was  initially  developed  that  emphasizes 
the  cognitive  aspects  of  team  collaboration  and  includes  the  major  human  decision-making 
processes  used  during  team  collaboration  (Warner,  Letsky  &  Cowen,  2005).  An  overarching 
objective  of  the  research  reported  here  is  to  test  the  current  coding  scheme  which  is  included  in 
the  model.  Schemes  for  coding  communications  data  should  be  mutually  exclusive,  exhaustive, 
and  equivalent. 

Our  previous  research  indicates  a  non-exhaustive  set  of  macrocognitive  processes  in  the  model 
of  team  collaboration.  Decision  making  -  what  we  label  decision  to  take  action  (DTA)  - 
emerged  during  previous  analyses  of  six  task  domains  as  a  new  macrocognitive  process, 
indicating  decision  making  is  an  essential  macrocognitive  process  when  teams  are  involved  in 
responding  to  many  complex,  real-world  tasks.  Deciding  to  take  action  is  viewed  as  both  a 
macrocognitive  process  and  a  product  of  team  collaboration  (Klein,  1993).  Many  real-world 
tasks  require  making  decisions  over  the  course  of  the  entire  event  -  to  accomplish  the  work,  as 
opposed  to  the  team  making  one  final  decision  at  the  conclusion  of  the  scenario.  A  decision  can 
be  defined  as  a  “mental  event  that  occurs  at  a  singular  point  in  time. .  .that  leads  immediately  or 
directly  to  action”  (Hoffman  &  Yates,  2005,  p.  77).  From  this  perspective,  a  decision  is  a 
commitment  to  a  course  of  action.  Our  analysis  of  a  range  of  task  domains  indicates  the  types  of 
tasks  described  by  this  model  typically  involve  team  members  making  decisions  as  part  and 
parcel  of  the  team’s  collaborative  problem  solving. 

We  maintain  that  decision  making  is  an  essential  component  of  team  collaboration  for  effective 
team  problem  solving  when  the  team  is  performing  the  task  as  opposed  to  planning  for  a  task  to 
be  executed  in  the  future.  This  has  been  a  consistent  finding  across  six  task  environments 
analyzed:  firefighters  on  Sept  1 1  responding  to  the  attack  on  the  world  trade  center  (Hutchins, 
Bordetsky,  Kendall,  &  Garrity,  2007),  air  warfare  teams  on  a  Navy  ship  (Hutchins,  Kendall,  & 
Bordetsky,  2010a),  a  boarding  team  conducting  maritime  interdiction  operations  (Hutchins  & 
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Kendall,  2008);  NORAD  collaborating  with  the  FAA  on  Sept  1 1  to  ground  all  commercial  air 
traffic  (Hutchins  &  Kendall,  2010b),  an  Air  Force  team  responding  during  a  time-sensitive 
targeting  scenario  (Hutchins  &  Kendall,  2010c),  and  during  an  experiment  involving  UAV  real¬ 
time  planning  and  execution  (Hutchins  &  Kendall,  2010b).  Based  on  this  consistent  pattern  of 
findings  we  have  added  decision  to  take  action  to  our  coding  scheme. 

Team  Collaboration  Tasks 

The  types  of  problem-solving  situations  accounted  for  by  the  model  of  team  collaboration  (Fiore, 
et  al,  2010)  are  ill-structured  decision-making  tasks  that  are  characterized  by  time  pressure, 
dynamically  changing  conditions,  ambiguous  and  incomplete  infonnation,  high  uncertainty  and 
high  cognitive  work-load  (that  is,  a  large  amount  of  knowledge  is  brought  to  bear  to  solve 
complex  problems),  as  well  as  human-system  interface  complexity.  Information  from  multiple 
sources  needs  to  be  considered  requiring  collaborative  analysis  to  ensure  team  members  come  to 
a  shared  understanding  of  the  task,  the  environment,  and  reach  consensus  on  a  course  of  action. 

Haiti  Humanitarian  Assistance/Disaster  Relief  (HA/DR)  Operation 

A  wide  range  of  diverse  organizations  partnered  -  including  non-governmental  organizations 
(NGOs),  various  military  commands,  Department  of  Defense  agencies,  public  and  private  relief 
agencies  and  non-profit  organizations  -  during  the  Haiti  relief  operation  to  fonn  an  alliance  to 
cooperate,  share  infonnation,  and  work  together.  Interorganizational  collaboration  has  been 
defined  as  “the  capability  of  an  organization  (or  set  of  organizations)  to  enter  into,  develop,  and 
sustain  interorganizational  systems  in  pursuit  of  collective  outcomes”  (Jansen,  Hocevar,  Rendon, 
&  Thomas,  2009,  p.  330).  As  described  by  Jansen  et  al  (2009),  the  focus  of  this  interorganiza¬ 
tional  collaboration  is  to  accomplish  a  wide  range  of  complementary  or  common  goals  and 
objectives,  including,  in  this  case,  the  common  overall  mission  of  providing  humanitarian 
assistance  and  disaster  relief.  The  extent  of  destruction  in  Port-au-Prince  made  rescue  work 
difficult,  and  with  many  surrounding  roads  impassable  and  airports  inoperable,  the  first 
shipments  of  foreign  aid  could  not  reach  the  people  who  most  needed  them. 

Measurement  Model  of  Team  Collaboration 

The  primary  purpose  of  this  paper  is  to  report  on  research  conducted  to  empirically  evaluate  and, 
if  necessary,  refine  a  model  of  team  collaboration  developed  by  Fiore  et  al  (2010).  This  model  of 
team  collaboration  has  evolved  over  the  course  of  the  CKI  research  program  and  the  most  recent 
version  of  the  model  is  depicted  in  Figure  1 .  This  revised  version  of  the  model  was  developed 
with  an  emphasis  on  the  ability  to  measure  the  macrocognitive  processes  that  transpire  during 
team  collaborative  problem  solving  in  a  laboratory  setting.  While  the  traditional  cognitive 
science  approach  has  been  to  focus  on  individual-level  constructs  such  as  perception,  attention, 
and  memory,  the  Fiore  et  al.  (2010)  model  emphasizes  ways  to  reliably  measure  what  team 
members  are  thinking  as  they  interact. 
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Figure  1.  Model  of  Team  Collaboration  (From  Fiore  et  al.,  2010). 


Definitions  of  the  macrocognitive  processes  (Fiore  et  al.,  2010)  included  in  the  model  are  listed 
in  Table  1,  which  also  includes  examples  from  the  APAN  Haiti  HA/DR  event  data  of  thought 
units  coded  as  representing  these  macrocognitive  processes.  A  thought  unit  refers  to  a  “sequence 
of  a  few  words  conveying  a  single  thought”  (Welden,  Jehn,  &  Pradhan,  1991,  p.  559),  or  “the 
smallest  message  unit  that  can  stand  alone”  (Keyton  &  Beck,  2010,  p.  336). 


Table  1.  Macrocognitive  Process  Definitions  used  for  coding  with  Examples  from  the 

Haiti  APAN  data. 1 


Stage  I:  Individual  Knowledge  Building  Process:  Actions  individuals  engage  in  to  add  to  their  existing  knowledge 
such  as  reading,  asking  questions,  accessing  information  on  displays,  providing  messages  to  team  members. 
Individual  Information  Synthesis:  Involves  comparing  relationships  among  information,  context,  and 
artifacts  to  develop  actionable  knowledge  or  affordances  (e.g.,  pushpins  and  attached  notes). 

-  No  coded  examples  for  APAN  data 

Individual  Information  Gathering:  Involves  actions  individuals  engage  in  to  add  to  their  existing  knowledge 
such  as  reading,  asking  questions,  accessing  displays,  providing  messages  to  teammates. 

•  Does  anyone  know  if  the  following  docks  located  outside  the  general  port  area  are  functional? 
Knowledge  Object  Development:  Involves  creation  of  artifacts,  by  the  individual,  that  represents  actionable 
knowledge  or  affordances  (e.g.,  pushpins,  attached  notes). 

-  No  coded  examples  for  APAN  data 

Stage  III:  Team  Knowledge  Building  Process:  Includes  actions  taken  by  teammates  to  disseminate  information 
and  to  transform  that  information  into  actionable  knowledge  for  team  members. 
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Team  Information  Exchange:  Involves  passing  relevant  information  to  the  appropriate  teammates  at  the 
appropriate  times.  Relevant  information  is  information  that  is  useful  in  helping  solve  a  particular  task. 

•  “NAME”  (USN  Retired)  has  250  water  purification  devices  available  in  Puerto  Plata,  Dominican 

Republic,  available  for  transfer  to  Port-Au-Prince. 

•  I  think  the  main  thing  I’d  recommend  is  that  the  NGOs  you  are  speaking  to  get  their  needs  communicated 
back  into  the  system. 

Team  Knowledge  Sharing:  Involves  explanations  and  interpretations  shared  between  team  members  or  with 
the  team  as  a  whole. 

•  SOUTHCOM  has  an  ACOE  [Army  Core  of  Engineers]  engineer  team  which  did  an  assessment  of  the  port 
of  Port-Au-Prince,  and  they  have  determined  that  the  entire  Port-Au-Prince  is  unusable  for  large  vessels. 

•  We  can  use  trucks  with  lights  and  have  ambulances  available  to  transport  patients  a  few  hundred  yards  to 
hospital. 

Team  Solution  Option  Generation:  Describes  offering  potential  solutions  to  the  problem. 

•  Killick  CG  base  may  possibly  be  used  for  offloading  supplies,  however,  limited  to  small  vessels  estimated 
47  ft  or  less  shallow  draft. 

•  Tactical  solution  -  use  5k  and  10k  cargo  nets  as  slings  under  rotary  wing  to  deliver  supplies  to  1DP 
[indigenous  displaced  personnel]  areas  and  distribution  points. 

Team  Evaluation  and  Negotiation  of  Alternatives:  Clarifying  and  discussing  the  pros  and  cons  of  potential 
solution  options.  This  could  include  clarifying  pieces  of  information,  verbally  simulating  the  ripple  effects  of 
alternatives,  attempting  to  persuade  other  teammates  regarding  the  relative  efficacy  of  alternatives. 

-  No  coded  examples  for  APAN  data 

Team  Process  and  Plan  Regulation:  Discussing  or  critiquing  the  team’s  knowledge  building  process  or  plan. 

-  No  coded  examples  for  APAN  data 

Stage  III:  Internalized  Team  Knowledge  Products: 

Teammate  Knowledge  Similarity:  The  degree  to  which  teammates’  mental  models  of  one  another’s 
relatively  stable  levels  of  skill,  knowledge,  experience,  dispositions  and/or  habits  converge. 

-  No  coded  examples  for  APAN  data 

Shared  Situation  Awareness:  The  degree  to  which  teammates’  awareness  and  interpretation  of  moment-by¬ 
moment  changes  in  their  collective  situation  converges. 

-  No  coded  examples  for  APAN  data 

Team  Knowledge  Objects:  Creation  of  artifacts  that  represent  actionable  knowledge  or  affordances  agreed  to 
by  the  team. 

-  No  coded  examples  for  APAN  data 

Task  Knowledge  Stock:  Accurate  task-relevant  knowledge  held  by  team  members.  This  would  include 
knowledge  about  task  strategy  and  equipment. 

-  No  coded  examples  for  APAN  data 

Stage  IV:  Externalized  Team  Knowledge  Products:  Refers  to  facts,  relationships,  and  concents  that  have  been 
explicitly  agreed  upon,  or  not  openly  challenged  or  disagreed  upon,  by  factions  of  the  team. 

Uncertainty  Resolution:  The  degree  to  which  a  team  has  collectively  agreed  upon  the  status  of  problem 
variables  (e.g.,  hostile/friendly). 

-  No  coded  examples  for  APAN  data 

Externalized  Cue-strategy  Associations:  Describes  the  team’s  collective  agreement  as  to  their  task  strategies 
and  the  situational  cues  that  modify  those  strategies  (and  how). 

-  No  coded  examples  for  APAN  data 

Pattern  Recognition  and  Trend  Analysis:  The  accuracy  of  the  patterns  or  trends  explicitly  noted  by  members 
of  a  team  that  is  either  agreed  upon  or  unchallenged  by  other  team  members. 

-  No  coded  examples  for  APAN  data 

Stage  V:  Team  Problem  Solving  Outcomes:  Assessments  of  duality  relating  to  a  team’s  problem  solutions 
or  plan. 

Quality  of  Plan/  Problem  Solving  Solution:  The  degree  to  which  the  solution  adopted  by  a  problem  solving 
team  achieves  a  resolution  to  the  problem  (e.g.,  limit  fatalities,  limit  destruction). 

6 


-  No  coded  examples  for  APAN  data 

Efficiency  of  Planning  Process:  Efficiency  of  planning  process  describes  the  amount  of  time  it  takes  a 
problem  solving  team  to  arrive  at  a  successful  resolution  to  a  problem. 

-  No  coded  examples  for  APAN  data 

Efficiency  of  Plan  Execution:  Describes  the  quality  of  the  plan  (e.g.,  number  of  lives  saved)  divided  by  the 
amount  of  resources  used  to  accomplish  this  and  the  amount  of  time  the  plan  takes  to  arrive  at  a  successful 
resolution  to  a  problem. 

-  No  coded  examples  for  APAN  data 

(Definitions  of  macrocognitive  processes  from  Fiore  et  al.,  2010.) 

Our  previous  analysis  of  several  task  domains  indicates  that  several  of  the  macrocognitive 
processes  included  in  the  model  of  team  collaboration  cannot  be  measured  during  certain  types 
of  real-world  team  collaborative  problem-solving  situations.  The  lack  of  evidence  for  several 
macrocognitive  processes  is  attributed  to  two  explanations. 

The  first  is  that  in  certain  types  of  problem-solving  environments,  such  as  task  domains  that 
require  dynamic  decision  making  (Montgomery,  1993;  Montgomery,  1989),  teams  do  not 
employ  some  of  the  processes  included  in  the  model  due  to  the  rapid  responses  required  to  deal 
with  emergent  events.  For  example,  Klein  (1989)  found  that  when  decision  makers  use  a 
recognition-primed  decision-making  strategy,  usually  the  situation  itself  either  detennines  or 
constrains  the  response  options  and  that  experienced  decision  makers  make  up  to  90%  of  all 
decisions  without  considering  alternatives.  If  the  situation  appears  similar  to  one  that  the  decision 
maker  has  previously  experienced,  the  pattern  will  be  recognized  and  the  course  of  action  is 
usually  immediately  obvious. 

Firefighters,  as  well  as  team  members  in  many  other  domains  collaborate  on  the  ‘front  end’  of 
the  problem  -  that  is  during  situation  assessment  and  subsequent  development  of  situation 
awareness,  but  typically  do  not  collaborate  on  how  to  respond  to  an  event  due  to  time  pressure. 
For  example,  we  saw  little  evidence  for  team  evaluation  and  negotiation  of  alternatives  by  team 
members  during  dynamic  decision-making  problem-solving  task  domains  because  many 
responses  are  guided  by  standard  operating  procedures,  and  the  stored  schemas  of  highly 
experienced  operators. 

A  second  reason  our  analyses  have  consistently  provided  a  lack  of  evidence  for  some  macrocog¬ 
nitive  processes  included  in  the  model  is  that  the  processes  included  in  the  Internalized  Team 
Knowledge  Products  Stage  (stage  III,  in  Table  1)  require  direct  interaction  with  team  members  to 
gather  data  required  to  measure  certain  processes.  These  cognitive  processes  -  Teammate 
Knowledge  Similarity,  Shared  Situation  Awareness,  and  Task  Knowledge  Stock,  are  amenable  to 
measurement  in  laboratory  settings  but  it  is  typically  not  feasible  to  obtain  measurements  on 
participants  during  a  complex  high-fidelity  exercise  or  during  a  real-world  event.  Our  focus  has 
been  on  analyzing  transcripts  or  chat  logs  obtained  after  the  real-world  event  has  transpired,  such 
as  the  transcript  from  the  Fire  Department  of  New  York  on  September  11,  2001,  and  the  Haiti 
HA/DR  effort  reported  on  in  this  paper. 

In  a  similar  vein,  Team  Problem  Solving  Outcomes  (stage  V,  in  Table  1)  requires  performance 
assessment  metrics  to  assess  the  Quality  of  the  Plan/Problem  Solving  Solution,  Efficiency  of 
Planning  Process,  and  Efficiency  of  Plan  Execution.  Performance  metrics  for  the  types  of  task 
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domains  we  have  studies  o  not  exist.  Measurement  strategies  to  assess  how  the  team  performed 
overall,  such  as  quality  of  the  plan/problem  solving  solution,  and  efficiency  of  planning  process, 
are  not  available  and  developing  these  types  of  measures  would  require  a  major  effort  in  itself. 

Additional  Macrocognitive  Processes  used  for  Analysis 

A  consistent  pattern  was  evident  across  the  task  domains  we  previously  investigated  where  the 
majority  of  thought  units  were  coded  as  representing  two  of  the  cognitive  processes:  T earn 
Information  Exchange  (TIE)  and  Team  Knowledge  Sharing  (TKS).  The  percentage  of  communi¬ 
cations  coded  as  TIE  and  TKS,  in  previously  reported  data,  and  for  the  current  data,  are  shown  in 
Table  2.  The  high  percentage  of  communications  coded  as  TIE  motivated  us  to  attempt  to  unpack 
this  category  by  investigating  if  other  cognitive  processes  might  be  occurring. 

Table  2.  Percentage  of  Communications  Coded  as  Team  Information  Exchange  and 
Team  Knowledge  Sharing  across  Task  Domains. 


Task  Domain 

Team  Information 
Sharing 

Team  Knowledge 
Exchange 

Maritime  Interdiction  Operation 

60. 01 

5.02 

Air  Operations  Center 

51.51 

10.32 

NORAD 

52.91 

3.72 

UAV  Dynamic  Re -planning 

58.00 

4.29 

Haiti  HA/DR  Effort 

81.00 

5.90 

Note:  In  a  previous  version  of  the  model  this  macrocognitive  process  was  labeled  Team  Information  Exchange. 
2 

In  a  previous  version  of  the  model  this  macrocognitive  process  was  labeled  Team  Knowledge  Sharing. 


Since  we  are  focusing  on  tasks  performed  in  “the  wild”  we  culled  from  the  naturalistic  decision¬ 
making  (NDM)  literature  related  to  macrocognition  to  develop  an  additional  list  of  macrocog¬ 
nitive  processes  that  could  be  used  for  coding  the  present  set  of  data.  Based  on  a  review  of  the 
macrocognition  literature  related  to  NDM  we  elected  to  include  additional  macrocognitive 
processes  in  a  revised  coding  scheme.  (More  detail  is  provided  in  the  Method  section.)  These 
macrocognitive  processes  and  their  definitions  listed  in  Table  2. 

Sensemaking.  Sensemaking  is  an  essential  cognitive  function  performed  by  a  variety  of  domain 
practitioners  across  a  wide  range  of  real-world  tasks  (Klein,  Phillips,  Rail,  and  Peluso,  2007). 

The  sensemaking  process  begins  when  a  person  becomes  aware  of  a  weakness  in  their  current 
comprehension  of  a  situation,  often  experienced  as  a  surprise,  in  response  to  unexpected  changes 
or  as  a  failure  of  expectations.  Sensemaking  is  a  critical  process  for  teams  engaged  in  real-world 
domains  were  practitioners  deal  with  complex,  dynamic,  evolving  situations  that  are  “rich  with 
various  meanings”  (Klein,  et  al,  2007,  p.  1 14).  The  data  these  practitioners  use  to  develop  an 
understanding  of  the  situation  are  often  highly  ambiguous  and  very  complex  and  the  dynamic 
events  require  the  decision  maker  to  dynamically  update  their  understanding  as  the  situation 
evolves  over  time.  The  frame  that  is  adopted  by  the  practitioner  will  affect  what  data  are  attended 
to  and  how  these  data  items  are  interpreted.  When  the  practitioner  notices  data  that  do  not  fit  the 
current  frame,  the  sensemaking  cycle  of  continuously  moving  toward  better  explanations  is 
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activated.  Sensemaking  incorporates  consideration  of  the  following  criteria:  plausibility, 
pragmatics,  coherence,  and  reasonableness  (Klein,  et  al,  2007). 

Anticipatory  thinking  is  described  as  a  critical  macrocognitive  function  of  both  individuals  and 
teams  (Klein,  Snowden,  &  Lock  Pin  (2007).  Anticipatory  thinking  is  form  of  sensemaking  that  is 
future  oriented  such  as  forming  expectancies  about  future  events  (e.g.,  Weick  &  Sutcliffe,  2001). 
Active  attention  management  where  the  operator  focuses  attention  on  likely  sources  of  critical 
information  is  a  key  characteristic.  Assessing  Risk  refers  to  an  evaluation  of  the  potential 
consequences  for  risk  or  a  danger  to  reach  a  desired  end-state.  Problem  Detection  is  defined  as  a 
form  of  sensemaking  that  recognizes  issues  arising  from  the  current  situation  and  the  outcome 
that  could  be  detrimental  if  not  addressed.  Using  Analogs  refers  to  comparing  the  current 
situation  with  past  experiences  to  solve  the  current  situation.  Planning  and  Re-planning  involves 
a  process  where  team  members  build  a  list  of  actions  that  will  be  performed  to  solve  a  problem 
and  adjust  as  developments  occur. 

Table  3.  Macrocognitive  Processes  from  the  Naturalistic  Decision-Making  Literature. 


Macrocognitive  Process 

Definition 

Anticipatory  Thinking 
(AT) 

To  foresee  and  make  mental  preparation  by  employing  one’s  mind  to  a 
hypothetical  scenario  or  problem  (contingency  preparation) 

Assessing  Risk 

(AR) 

Evaluation  of  the  potential  consequences  for  risk  or  a  danger  to  reach  a  desired 
end-state 

Problem  Detection 
(PD) 

A  form  of  sensemaking  that  recognizes  issues  arising  from  the  current  situation 
and  the  outcome  that  could  be  detrimental  if  not  addressed 

Planning  and  Re-planning 

(PR) 

A  process  where  team  members  build  a  list  of  actions  that  will  be  performed 
to  solve  a  problem  and  adjust  as  developments  occur 

Sensemaking 

(SM) 

The  process  of  framing  and  reframing  current  inputs  to  the  problem  in  a 
continuous  process  that  helps  us  filter  and  interpret  the  data 

Using  Analogues 
(UA) 

Comparing  the  current  situation  with  past  experiences  to  solve  the 
current  situation 

In  the  following  sections  we  describe  the  collaboration  tool  that  was  used  during  the  Haiti 
HA/DR  effort  -  All  Partners  Area  Network  (APAN)  -  and  a  method  for  the  analysis  of  text  files 
-  lexical  link  analysis  (LLA).  Use  of  the  LLA  results  provided  a  systematic  way  to  select  a 
subset  of  the  large  amount  of  APAN  data  from  the  Haiti  HA/DR  effort  for  our  subsequent 
analysis.  We  selected  several  themes  that  were  identified  by  LLA  -  the  ‘water’  and  ‘hospital’ 
themes  -  to  analyze  from  initiation  to  completion. 

All  Partners  Area  Network  Collaboration  Tool 

The  All  Partners  Area  Network  (APAN)  is  an  internet-based  tool  which  was  designed  to  facile- 
tate  infonnation  sharing  between  participants  during  disaster  relief  efforts  and  was  used  in  the 
Haiti  relief  effort.  APAN  is  a  web  site  that  combines  the  benefits  of  unstructured  collaboration 
(wikis,  blogs,  forum)  and  structured  collaboration  (file  sharing,  calendar)  with  social  networking 
to  facilitate  information  sharing  with  multinational  partners,  NGOs,  and  US  Federal  and  State 
agencies.  Over  1700  different  individuals  utilized  the  site  during  the  relief  effort  (Pierce,  2010). 
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APAN  has  recently  been  revised  and  the  version  used  in  the  Haiti  relief  operation  was  the 
Transnational  Information  Sharing  Cooperation  (TISC).  This  version  of  APAN  was  created  in 
2008  to  meet  the  needs  of  the  US  Pacific  Command  (PACOM),  who  oversees  a  vast  operational 
area  with  many  different  countries.  The  goal  of  this  tool  is  to  facilitate  the  free  flow  of  informa¬ 
tion  between  relief  organizations  and  the  military  throughout  PACOM’s  Area  of  Responsibility 
(AOR)  (Ives,  2010).  In  particular,  the  site  was  to  be  used  in  areas  without  sophisticated  technical 
capabilities  -  a  working  internet  connection  is  all  that  is  required  to  use  the  site. 

The  simplicity  of  the  site  was  one  of  the  major  reasons  for  its  adoption  among  relief  workers. 

The  site  was  designed  to  be  easy  to  use  in  order  to  facilitate  use  by  personnel  who  may  not  be 
technologically  savvy  (Pierce,  2010).  Another  benefit  of  this  tool  is  that  it  enables  relief 
providers  to  advertise  their  services  allowing  the  user  to  pick  and  chose  what  they  need.  An 
example  of  this  was  with  the  Sacre  Coeur  hospital  in  Milo,  Haiti.  The  hospital  was  not  damaged 
in  the  earthquake,  but  many  people  did  not  know  this  and  thus  the  hospital  was  severely 
underutilized.  The  hospital  admitted  only  6  people  in  the  four  days  following  the  disaster. 
Hospital  personnel  announced  the  availability  of  beds  on  APAN  and  the  US  military  responded 
by  airlifting  severely  injured  patients  to  the  hospital  and  increasing  its  hospital  admittance  to  250 
within  a  few  days  (Technology  Links  Medical  Aid  and  Survivors,  2010). 

Lexical  Link  Analysis 

Lexical  link  analysis  (LLA),  a  tool  for  the  analysis  of  structured  or  unstructured  text  files,  based 
on  an  automated  parsing  of  documents,  to  develop  themes  for  the  examined  database  based  on 
the  frequency  of  occurrence  of  key  words  (Gallup  &  Wood,  2010).  These  themes  are 
characterized  by  one,  two,  or  three  key  words,  where  themes  such  as  ‘water’,  ‘hospital-supplies’, 
or  ‘injuries’  are  thus  discovered.  The  LLA  is  composed  of  two  components,  lexical  analysis  and 
link  analysis.  Lexical  analysis  is  a  form  of  text  mining  in  which  word  pairs  are  extracted  from  a 
specific  set  of  documents,  such  as  Blog  and  Forum.  Link  analysis  is  a  subset  of  network  analysis 
that  explores  associations  between  objects,  revealing  the  crucial  relationships  between  objects. 
Link  analysis  “discovers”  and  displays  a  network  of  word  pairs. 

A  LLA  was  performed  on  the  APAN  data  to  identify  themes  that  emerged  and  what  organiza¬ 
tions  were  involved  in  each  theme  over  the  course  of  the  Haiti  HA/DR  operation  (Zhao,  2010). 
The  APAN  data  was  from  a  Microsoft  Structured  Query  Language  (MSSQL)  database  and  the 
sources  that  were  analyzed  included:  (i)  Official  documents  and  briefings:  167  PDF  file 
attachments  related  to  HAITI  HA/DR  from  1/13/2010  to  5/26/2010;  (//')  SITREP:  150  Situation 
Report  documents;  (///)  Forum:  1 173  posts  from  1/13/2010  to  6/3/2010;  and,  (z'v)  Blogs:  3900 
blog  messages. 

Forty  themes  were  generated  by  identifying  noun-verb  word  pairs  where  noun  and  verbs  next  to 
each  other  were  found  in  documents.  Weights  were  assigned  to  word  pairs  according  to  themes 
and  documents  for  each  day.  Results  of  the  LLA  provided  a  systematic  way  to  select  a  subset  of 
the  huge  amount  of  APAN  data  for  analysis  for  this  research.  For  example,  all  information 
exchanges  that  concerned  “water”  were  tagged  as  related  to  the  “water”  theme  and  were 
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subsequently  analyzed  from  the  initiation  of  that  theme  through  the  conclusion  of  information 
exchanges  pertaining  to  water. 


METHOD 

The  ONR  CKI  Program  Manager  was  provided  a  complete  set  of  the  APAN  data  from  the  Haiti 
HA/DR  Effort.  ONR  went  through  its  institutional  review  board  (IRB)  process  to  obtain 
pennission  to  use  this  data  and  to  distribute  the  data  to  researchers  conducting  under  the  CKI 
program.  We  also  went  through  the  Naval  Postgraduate  School  IRB  process  to  obtain  approval 
for  use  of  this  data. 

Selecting  Data  for  Analysis 

Data  representing  two  themes  were  selected  based  on  the  results  of  the  LLA  -  ‘water’  and 
‘hospital’,  and  all  communications  associated  with  these  themes  were  analyzed  and  coded  to 
empirically  evaluate  the  measurement  model  of  team  collaboration.  This  step  served  two 
purposes.  First,  it  provided  a  systematic  way  to  select  a  subset  of  the  vast  amount  of  data 
included  in  the  overall  APAN  data  set,  which  includes  communications  between  some  1700 
responders  over  a  5 -month  time  period.  Second,  it  also  provided  a  way  to  isolate  all  communi¬ 
cations  related  to  a  particular  topic,  such  as  ‘hospital,’  ‘water,’  or  ‘security.’  We  analyzed  the 
data,  using  definitions  of  the  macrocognitive  processes  included  in  the  model  of  team 
collaboration  to  gain  insight  into  the  inter-agency  collaboration  that  transpired  between  the 
various  responders  who  represented  a  large  number  of  organizations  during  this  large-scale 
effort. 

Data  Analysis  and  Coding 

One  pair  of  coders,  as  part  of  the  research  for  their  master’s  thesis,  analyzed  two  themes;  each 
coder  analyzed  and  coded  all  thought  units  associated  with  the  water  theme  and  the  hospital 
theme.  A  thought  unit  refers  to  a  sequence  of  words  that  convey  a  single  thought  (Welden  et  al, 
1991). Cohen’s  kappa  coefficient  was  used  to  calculate  the  percentage  of  agreement  between  the 
two  coders. 

Organizing  data.  The  APAN  data  was  originally  from  a  MSSQL  database  and  was  exported  as 
excel  spreadsheets  by  NPS  students.  Data  included  people's  names,  organizations,  email 
addresses,  etc.  Investigators  replaced  all  personally  identifiable  information  (PII)  by  substituting 
all  personal  identifiers  of  chat/blog  respondents  when  the  data  set  was  received.  Prior  to  coding, 
the  data  was  unitized  by  separating  each  thought  units  on  a  separate  line  in  the  excel  spreadsheet. 
Each  thought  unit  was  given  a  separate  code.  A  thought  unit  is  not  equivalent  to  a  sentence,  as 
one  sentence  may  contain  several  thought  units  and  several  sentences  may  contain  one  thought 
units. 

Practice  coding.  Students  independently  practiced  coding  200  lines  from  a  separate  data  set  to 
gain  experience  in  applying  the  definitions  of  the  macrocognitive  processes  included  in  the 
model  of  team  collaboration.  Raters  discussed  their  respective  coding  with  the  lead  researcher  to 
calibrate  their  use  of  the  macrocognitive  process  categories.  They  then  completed  additional  practice 
coding  on  200  lines  of  APAN  data  -  on  a  separate  theme  -  prior  to  their  coding  the  data  reported 
here  to  ensure  they  were  calibrated.  Following  this  training  period  they  independently  coded  the  data 
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from  the  forum  and  blogs  related  to  the  water  and  hospital  themes,  subsequently  reviewed  their  coding, 
calculated  percent  agreement,  and  resolved  any  differences  in  coding. 

Inter-Rater  Reliability.  Both  coders  coded  682  and  538  thought  units  for  the  ‘water’  and 
‘hospital’  themes,  respectively.  Cohen’s  kappa  coefficient  indicated  high  levels  of  agreement 
between  coders  for  both  sets  of  data.  Inter-rater  agreement  was  72.0%  and  70.0%,  for  the  water 
theme  and  the  hospital  themes,  respectively. 

New  Macrocognitive  Processes 

As  discussed  in  the  background  section,  we  were  motivated  to  unpack  the  macrocognitive 
category  that  contained  the  majority  of  team  communications,  Team  Information  Exchange 
(TIE).  The  research  team  reviewed  the  NDM  literature  related  to  macrocognition  to  determine 
additional  cognitive  processes  that  might  be  used  to  differentiate  the  large  number  of 
communications  that  were  coded  as  TIE.  Macrocognitive  processes  included  in  our  coding 
scheme  included  sensemaking,  anticipatory  thinking,  problem  detection,  assessing  risk,  planning 
and  re-planning,  and  using  analogues.  After  reviewing  the  definitions  found  in  the  literature  we 
arrived  at  a  consensus  on  definitions  for  the  macrocognitive  processes  listed  and  defined  in  Table 
2  (in  the  background  section),  and  they  were  added  to  the  list  of  macrocognitive  processes  to  be 
used  for  analyzing  and  coding  the  APAN  data. 

RESULTS 

Table  4  presents  the  percentage  of  communications  between  aid  workers  in  the  water  and  hospital 
themes  coded  as  representing  the  macrocognitive  processes  in  the  measurement  model  of  team 
collaboration.  The  pattern  of  results  is  strikingly  similar  for  both  themes  and  is  similar  to  results 
obtained  during  previous  use  of  the  coding  scheme  in  the  measurement  model  of  team 
collaboration.  As  can  be  seen,  the  vast  majority  of  team  members’  communications  concerned 
knowledge  construction,  that  is,  Team  Knowledge  Building  -specifically  team  information 
exchange,  highlighting  the  importance  of  this  process  during  the  Haiti  relief  effort.  Team 
Information  Exchange,  and  to  a  lesser  degree,  Team  Knowledge  Sharing,  and  Team  Solution 
Option  Generation  are  used  by  team  members  to  develop  an  understanding  of  the  complex 
problem  and  to  refine  and  maintain  their  understanding  as  the  highly  dynamic  situation  evolves. 
Of  particular  interest  is  the  large  number  of  speech  turns  coded  as  Team  Information  Exchange 
(tie):  TIE  involves  passing  relevant  information  to  the  appropriate  teammates  at  the  appropriate 
times.  Relevant  information  is  information  that  is  useful  in  helping  solve  a  particular  task. 


Table  4.  Percentage  of  Communications  Coded  as  Macrocognitive  Processes. 


Code 

Macrocognitive  Process  Categories 

Percentage  of  Thought  Units 

Individual  Knowledge  Building 

Water 

Hospital 

IIG 

Individual  Information  Gathering 

4.10 

3.60 

IIS 

Individual  Information  Synthesis 

0.00 

0.00 

KOB 

Knowledge  Object  Development 

0.00 

0.00 

Team  Knowledge  Building 

TIE 

Team  Information  Exchange 

81.00 

81.80 
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TKS 

Team  Knowledge  Sharing 

5.90 

5.60 

TSOG 

Team  Solution  Option  Generation 

1.80 

1.50 

TENA 

Team  Evaluation  and  Negotiation  of  Alternatives 

0.00 

0.00 

TPPR 

Team  Process  and  Plan  Regulation 

0.00 

0.00 

Internalized  Team  Knowledge 

ITK 

Teammate  Knowledge  Similarity 

0.00 

0.00 

SSA 

Shared  Situation  Awareness 

0.00 

0.00 

TKR 

Team  Knowledge  Objects 

0.00 

0.00 

IK 

Task  Knowledge  Stock 

0.00 

0.00 

Externalized  Team  Knowledge 

UR 

Uncertainty  Resolution 

0.00 

0.00 

ECS  A 

Externalized  Cue-strategy  Associations 

0.00 

0.00 

PRTA 

Pattern  Recognition  and  Trend  Analysis 

0.00 

0.00 

Problem  Solving  Outcomes 

QOP 

Quality  of  Plan/  Problem-solving  Solution 

0.00 

0.00 

EPP 

Efficiency  of  Planning  Process 

0.00 

0.00 

EPE 

Efficiency  of  Plan  Execution 

0.00 

0.00 

Decision  to  Take  Action 

DTA 

DTA  (Issue  Course  of  Action) 

0.00 

0.00 

RTA 

DTA  (Request  Take  Action) 

7.30 

7.40 

Team’s  Use  of  Additional  Macrocognitive  Processes 

It  is  noteworthy  that  the  third  most  frequently  used  cognitive  process  was  a  subcategory  of 
Decision  to  Take  Action,  that  is,  Request  Take  Action  (the  last  entry  in  Table  4),  reflecting  how 
decision  making  is  often  required  to  effectively  respond  during  a  real-world  event. 

Tables  5-8  present  examples  of  instances  where  thought  units  were  coded  with  one  of  the 
macrocognitive  processes  from  the  NDM  literature.  As  reflected  in  Table  5,  in  all  cases  the 
thought  unit  was  coded  using  one  of  the  initial  set  of  macrocognitive  processes  included  in  the 
model  of  team  collaboration  and  then  when  appropriate  was  coded  with  an  additional  category. 

In  the  vast  majority  of  cases  where  coders  assigned  a  second  code  these  thought  units  were 
initially  coded  as  team  information  exchange  (TIE).  For  example,  Table  5  provides  examples  of 
thought  units  coded  as  team  information  exchange  that  were  also  coded  as  anticipatory  thinking 
(AT).  In  the  first  example,  the  person  is  anticipating  the  need  for  additional  beds  for  patients  and 
is  letting  other  relief  workers  know  that  the  hospital  where  he  is  working  will  be  able  to 
accommodate  100  additional  patients  in  the  coming  days.  In  the  second  example  of  anticipatory 
thinking  the  relief  worker  informs  others  that  he  has  “perhaps  twelve  hours  of  working  materials 
to  keep  going,”  anticipating  a  shortage  of  necessary  material  that  same  day. 


Table  5.  Examples  from  the  ‘Hospital’  theme  coded  as  Anticipatory  Thinking. 


Thought  Unit 

Model 

Code 

NDM 

Code 

1. 

We  have  capacity  for  10  patients  immediately  and  probably  100  more  in  the  coming  days. 

TIE 

AT 

2. 

At  this  point,  like  1  was  saying,  we  have  perhaps  twelve  hours  of  working  materials  to 
keep  going. 

TIE 

AT 
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3. 

The  use  of  CAP  (combat  air  patrol)  will  avoid  the  logjam  at  PAP  [Port-au-Prince]  and  is 

TIE 

AT 

an  effective  innovation  in  putting  the  supplies  and  aid  where  they  are  needed. 

Note:  AT  =  Anticipatory  thinking 


Table  6  provides  examples  of  thought  units  coded  as  problem  detection  (PD)  from  the  hospital 
theme.  Problem  detection  refers  to  a  situation  where  the  person  recognizes  an  issue  arising  from 
the  current  situation  where  the  outcome  could  be  detrimental  if  not  addressed.  These  involve 
instances  where  the  person  is  informing  others  of  the  need  for  (1)  security,  (2)  contact  with  the 
only  operational  hospital,  (3)  medicine,  (4)  food  and  water,  (5)  security,  and  (6)  additional 
medical  care.  In  all  these  instances  of  problem  detection,  labeling  these  instances  of  team 
information  exchange  provides  a  greater  degree  of  specificity  beyond  team  information 
exchange. 


Table  6.  Examples  from  APAN  data  Hospital  theme  coded  as  Problem  Detection. 


Thought  Unit 

Model 

Code 

NDM 

Code 

1. 

Improvement  in  delivery  method  will  aid  some  and  can  be  implemented  immediately 
but  the  final  solution  will  require  security  on  the  ground. 

TIE 

PD 

2. 

Unfortunately  the  situation  is  critical  and  although  US  choppers  are  flying  overhead 
regularly  there  is  as  yet  no  contact  between  one  of  the  few  remaining,  standing 
hospitals  in  the  country  and  the  US  military. 

TIE 

PD 

3. 

We  have  run  out  of  antibiotics  and  analgesics. 

TIE 

PD 

4. 

500  children  40  miles  N/NE  of  Port-au-Prince  that  have  about  24  hours  of  food  and 
water  left! ! ! 

TIE 

PD 

5. 

The  rumor  is  that  force  protection  -  a  Force  protection  requirement  -  is  impeding 
aid  delivery. 

TIE 

PD 

6. 

Again,  we  have  over  a  thousand  patients  that  are  ready  for  surgery. 

TIE 

PD 

7. 

The  hospital  administrator  for  Bernard  Mevs  Hospital  located  near  the  Port-Au-Prince 
airport  has  reported  an  escalating  number  of  pediatric  malaria  cases  requiring  treatment. 

TIE 

PD 

8. 

The  hospital  official  noted  a  limited  supply  of  anti-malarial  pharmaceuticals  available. 

TIE 

PD 

9 

From  OCHA  [Office  for  the  Coordination  of  Humanitarian  Affairs]:  The  Haiti  emergency 
is  also  a  high-risk  environment  for  sexual  exploitation  and  abuse. 

TIE 

PD 

10. 

At  this  point,  like  I  was  saying,  we  have  perhaps  twelve  hours  of  working  materials 
to  keep  going. 

TIE 

PD 

11. 

In  these  operating  rooms,  we  don’t  have  oxygen,  we  don’t  have  general  anesthesia, 
we  don’t  have  narcotic  pain  medicines  in  enough  quantity. 

TIE 

PD 

12. 

Again,  we  have  over  a  thousand  patients  that  are  ready  for  surgery. 

TIE 

PD 

13. 

“WEBSITE”  does  not  seem  to  have  contacts  for  hospitals  in  Haiti  and  seems  to  provide 
maps  only  in  PDF  format. 

TIE 

PD 

14. 

I  saw  your  post  and  thought  perhaps  partners  may  know  of  someone  who  can  assist  us. 

IIG 

PD 

15. 

The  problem  we  are  facing  is  with  discharge. 

TIE 

PD 

Note:  PD  =  Problem  detection 


Table  7  provides  examples  of  thought  units  coded  as  problem  detection  (PD)  from  the  water 
theme. 
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Table  7.  Examples  from  APAN  data  Water  theme  coded  as  Problem  Detection. 


Thought  Unit 

Model 

Code 

NDM 

Code 

1. 

The  U.S.  has  supply  agreements  with  Luxembourg  (attached),  though  it  appears  they 
exclude  North  America. 

TIE 

PD 

2. 

At  this  moment  my  condition  is  very  difficult  with  the  orphans  because  we  can’t  have  food, 
water,  and  medicines  and  food  supplies  because  we  don’t  have  money  on  hand  to  do  that. 

TIE 

PD 

3. 

But  the  roads  are  not  yet  all  named  in  the  Montagne  Noire  area. 

TIE 

PD 

4. 

We  safely  dropped  shelter  boxes  (emergency  tent/filtration/blanket  survival  kits)  Friday 
but  gents,  this  is  food  and  I  fear  for  their  safety. 

TKS 

PD 

5. 

Last  contact  with  the  orphanage  revealed  little  to  no  food,  no  access  to  water  and  a  deep 
concern  about  possible  criminal  activity  in  the  area. 

TIE 

PD 

6. 

Jumping  the  water  from  US  to  Haiti  is  the  only  missing  piece  of  our  puzzle! 

TIE 

PD 

7. 

They  didn’t  have  radios,  batteries  or  generators  and  the  earthquake  silenced  landlines  and 
mobile  services,  leaving  Haitians  in  the  dark  in  more  ways  than  one. 

TIE 

PD 

8. 

MSC  or  airlift  needed  ASAP. 

TIE 

PD 

Table  8  provides  examples  of  thought  units  coded  as  Sensemaking  (SM)  from  the  hospital  theme. 


Table  8.  Examples  from  APAN  data  Hospital  theme  coded  as  Sensemaking. 


Thought  Unit 

Model 

Code 

NDM 

Code 

1. 

This  one  was  very  interesting  in  that  the  need  on  the  ground  if  Haiti  changed  based  on 
UNICEF  WASH. 

TIE 

SM 

3. 

The  comforting  doll  is  “Like  Them.” 

TIE 

SM 

4. 

Dr.  “name”:  This  question  of  security  and  the  rumors  of  security  and  the  racism 
behind  the  idea  of  security  had  been  our  major  block  to  getting  aid  in. 

TKS 

SM 

Note:  SM  =  Sensemaking 


Lastly,  an  example  of  using  analogies  (UA)  occurred  where  the  person  compared  the  current 
situation  in  Haiti  with  previous  HA/DR  efforts  where  “a  single  website  was  needed  to  support 
the  predictable  exchange  of  information  in  emergencies  at  the  country  level.”  A  second  example 
is  the  following:  “I  believe  there  is  such  hunger  that  food  brings  out  the  beast  in  any  uprooted 
survivor  of  a  12  day  old  disaster.” 

Generating  solutions  during  time-critical  tasks  is  an  important  aspect  of  collaboration.  Table  9 
provides  examples  of  thought  units  coded  as  Team  Solution  Option  Generation  (TSOG). 


Table  9.  Examples  of  Team  Solution  Option  Generation. 


Team  Solution  Options  Generated 

1. 

One  idea  might  be  to  set  up  Comms  between  the  hospital  and  the  USNS  Comfort  which  I  believe  will  be 
in  port  today. 

2. 

A  good  idea  perhaps  if  at  all  possible  is  have  someone  take  some  GPS  coordinates  of  the  landing  field 
and  post  it  here. 

3. 

My  driver  offered  to  wrap  every  medical  worker  in  5  Haitians  to  make  sure  they’d  feel  safe. 

4. 

Suggestion  for  military  to  consider:  Would  help  a  lot  if  military  came  across  as  focused  primarily  on 
humanitarian  aid,  less  on  security. 

15 


Inter-Rater  Reliability 

The  high  Cohen  kappa  coefficient  achieved  indicates  the  two  coder’s  agreement  is  substantial. 
This  high  inter-rater  reliability  indicates  the  macrocognitive  process  definitions  used  by  the 
coders  are  objective.  Table  10  presents  the  results  of  the  kappa  analysis,  for  the  water  theme 
data,  in  the  form  of  a  pivot  table  which  compares  coder  1  codes  with  coder  2  codes.  Coder  1 
codes  are  displayed  in  the  columns  and  coder  2  codes  are  read  across  rows.  The  diagonal  cells  of 
the  matrix  indicate  agreement  between  the  coders  whereas  the  values  in  the  other  cells  indicate 
the  difference  between  what  each  of  the  coders  chose.  The  pivot  table  also  highlights  which 
codes  the  coders  disagreed  upon.  For  example,  under  the  team  information  exchange  (TIE) 
category,  coder  1  and  coder  2  had  assigned  a  total  of  540  and  506  TIE  codes,  respectively,  to  the 
data.  However,  both  coders  matched  selections  for  488  of  the  TIE  codes. 


Table  10.  Pivot  Table  for  Coders  Analysis  of  Water  theme. 


Water  Cohen  Kappa  Results 

Row 

Labels 

O 

RTA 

TIE 

TKS 

O 

o 

02 

H 

c4 

D 

MISC 

u 

z 

TOTAL 

Coder  2 

IIG 

24 

0 

2 

0 

0 

0 

1 

0 

27 

RTA 

1 

34 

6 

1 

0 

0 

0 

0 

42 

TIE 

0 

10 

488 

6 

2 

0 

0 

0 

506 

TKS 

0 

0 

27 

19 

1 

0 

0 

0 

47 

TSOG 

0 

1 

7 

2 

5 

0 

0 

0 

15 

UR 

0 

0 

0 

0 

0 

0 

0 

0 

0 

MISC 

0 

0 

10 

0 

0 

0 

35 

0 

45 

NC 

0 

0 

0 

0 

0 

0 

0 

0 

0 

TOTAL 
Coder  1 

25 

45 

540 

28 

8 

0 

36 

0 

682 

Table  1 1  presents  the  results  of  the  Kappa  analysis,  for  the  hospital  theme  data. 

Table  11.  Pivot  Table  for  Coders  Analysis  of  Hospital  theme. 


H 

lospil 

tal  Cohen  Kappa  Results 

Row 

Labels 

O 

RTA 

TIE 

TKS 

TSOG 

at 

D 

MISC 

u 

z 

TOTAL 
CODER  2 

IIG 

13 

5 

3 

0 

0 

0 

0 

0 

21 

RTA 

0 

32 

8 

0 

0 

0 

0 

0 

40 

TIE 

1 

5 

380 

7 

0 

0 

0 

0 

393 

TKS 

0 

1 

18 

15 

0 

0 

0 

0 

34 

TSOG 

1 

2 

8 

0 

2 

0 

0 

0 

13 

UR 

0 

0 

0 

0 

0 

0 

0 

0 

0 

MISC 

0 

0 

8 

0 

0 

0 

29 

0 

37 

NC 

0 

0 

0 

0 

0 

0 

0 

0 

0 

TOTAL 
Coder  1 

15 

45 

425 

22 

2 

0 

29 

0 

538 
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DISCUSSION 


A  sequential  analysis  method  was  used  to  unpack  the  way  cognition  unfolded  over  the  course  of 
the  Haiti  HA/DR  operation  for  a  highly-dynamic,  complex,  large-scale  problem-solving  event. 
Several  of  the  macrocognitive  processes  included  in  the  model  cannot  be  captured  or  coded  with 
the  type  of  data  we  have  analyzed,  that  is,  a  set  of  communications  or  chat  log  entries  that 
occurred  in  a  real-world  setting,  but  were  received  after  the  event.  Since  the  data  we  focus  on  is 
data  captured  either  from  “the  wild”  or  in  high-fidelity  experiments,  we  have  no  ability  to 
measure  things  such  as  Teammate  Knowledge  Similarity,  Shared  Situation  Awareness,  Quality  of 
Plan/  Problem  Solving  Solution,  and  Efficiency  of  Planning  Process  because  they  require  (1) 
obtaining  specific  types  of  data  directly  from  team  members,  such  as  via  survey  administration, 
or  (2)  assessing  the  team’s  perfonnance,  which  often  is  not  possible  as,  in  many  cases,  there  are 
no  perfonnance  assessment  metrics  for  these  real-world  tasks. 

We  took  the  approach  of  adding  new  cognitive  processes  to  our  coding  scheme  in  an  effort  to 
discover  other  macrocognitive  processes  that  might  be  employed  by  collaborating  teams. 
Specifically  we  were  interested  in  cognitive  processes  that  are  currently  coded  as  Team 
Information  Exchange.  The  addition  of  macrocognitive  processes  from  the  NDM  literature  was 
an  exploratory  effort  to  discover  whether  additional  cognitive  processes  are  employed  during 
complex,  infonnation-rich,  problem-solving  events.  We  saw  evidence  for  these  additional 
processes,  that  is,  sensemaking,  anticipatory  thinking,  problem  detection,  assessing  risk,  planning 
and  re-planning,  and  using  analogues. 

Developing  a  more  comprehensive  understanding  of  the  macrocognitive  processes  involved  in 
team  collaboration  has  several  practical  advantages.  First,  conceptualizations  that  provide  the 
theoretical  foundation  for  a  model  of  team  collaboration  that  take  into  account  consistent 
findings  based  on  empirical  research  in  real-world  work  domains  are  likely  to  be  more  accurate. 
Second,  the  way  team  cognitive  processes  influence  team  functioning  was  listed  among  the  top 
ten  critical  research  questions  in  team  research  (Salas  &  Wildman,  2009).  Third,  a  more 
complete  model  of  team  collaboration  can  guide  designers  of  collaboration  tools  to  facilitate 
decision  making  as  part  of  the  overall  task. 

Decision  Making  is  Part  of  Problem-Solving 

Many  dynamic  and  evolving  tasks  that  involve  team  problem  solving  include  decision  making; 
that  is,  team  members  take  action,  or  request  other  teammates  to  take  action,  in  addition  to 
developing  new  knowledge  and  agreeing  on  a  final  solution.  Actions  are  frequently  part  of  the 
overall  information  gathering  process  and  have  diagnostic  functions  (Orasanu  &  Connolly, 

1993). 


CONCLUSIONS 

Developing  a  more  comprehensive  understanding  of  the  macrocognitive  processes  involved  in 
team  collaboration  has  several  practical  advantages.  First,  conceptualizations  that  provide  the 
theoretical  foundation  for  a  model  of  team  collaboration  that  take  into  account  consistent 
findings  based  on  empirical  research  in  real-world  work  domains  are  likely  to  be  more  accurate. 
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Second,  the  way  team  cognitive  processes  influence  team  functioning  was  listed  among  the  top 
ten  critical  research  questions  in  team  research  (Salas  &  Wildman,  2009).  Third,  a  more 
complete  model  of  team  collaboration  can  guide  designers  of  collaboration  tools  to  facilitate 
decision  making  as  part  of  the  overall  task. 

Decision  to  Take  Action  is  recommended  as  a  new  category  to  be  added  to  the  set  of  macrocogni- 
tive  processes  included  in  the  Fiore  et  al.  (2010)  model  of  team  collaboration.  Deciding  to  take 
action  is  viewed  as  both  a  macrocognitive  process  and  a  product  of  team  collaboration.  We 
assert  that  decision  making  is  a  critical  element  of  team  problem  solving  when  a  team  is 
executing  a  task,  in  contrast  to  conducting  a  planning  task.  When  team  members  collaborate  to 
solve  a  problem  they  make  decisions  and  implement  those  decisions  as  part  of  performing  the 
task. 

For  the  task  domains  discussed  here,  a  constant  interplay  exists  between  sharing  information  -  to 
develop  new  knowledge  and  maintain  situation  awareness — and  deciding  on  actions  that  are 
required,  and  implementing  those  actions,  followed  by  monitoring  the  situation  and  continuing  to 
build  new  knowledge  on  the  unfolding  situation.  Execution  of  the  mission,  or  problem-solving 
task,  would  come  to  a  screeching  halt  without  this  continual,  iterative  cycle  of  developing 
knowledge  of  the  situation  and  responding  to  the  current  situation. 
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Objective 

•  Evaluate  CKI  measurement  model  of  team  collaboration 


>  Based  upon  analysis  of  real-world  complex  decision-making  events 

>  Determine  which  macrocognitive  processes  are  used  and,  if  necessary, 
refine  the  model  based  on  empirical  analysis 

>  Develop  a  better  understanding  of  the  cognitive  processes  employed 
when  teams  collaborate  to  solve  problems 

•  Test  the  current  coding  scheme  included  in  the  model  of  team  collaboration 

-  Apply  definitions  of  macrocognitive  processes  included  in  CKI  model  of 
team  collaboration 

-  Schemes  for  coding  communications  should  be  mutually  exclusive, 
exhaustive,  and  equivalent 

•  Examine  a  range  of  real-world  task  domains 
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Method 


•  Macrocognition  -  emerging  field  within  cognitive 
engineering  that  describes  the  way  cognition 
occurs  in  naturalistic,  or  real-world,  decision¬ 
making  events  (Cacciabue  &  Hollnagell,  1995) 

•  Macrocognition  in  teams  expands  the  concept  by 
considering  group  cognition  and  the  collective 
(team)  cognitive  processes  that  enable  the 
externalization  of  internalized  knowledge  building. 

(Letsky  et  al.,  2008) 

•  Focus  on  cognition  in  collaboration  contexts 

-  Focuses  on  contextually-bound  processes 

-  Sensemaking,  managing  uncertainty,  and 
related  cognitive  processes  entailed  in 
responding  to  emerging  events  that  occur  in 
dynamic  decision-making  situations 

—  Teams  collaborate  on  short-term  situations  which 
require  rapid  action  to  be  taken  against  specific 
missions 

•  Content  analysis  -  analyze,  code  team  comm's 

•  Unique,  information-rich,  ambiguous,  time- 
compressed  scenarios  wwwnps.edu 


Problem  Area 

Characteristics  Collaboration  Stages  &  Cognitive  Processes 


Knowledge  Team  Team  Outcome  Eval.  Decision  to 

Construction  Problem  Solving  Consensus  &  Revision  Take  Action 


9:17 

9:18 

9:19 
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Analyzed  APAN  data  from  the  Haiti  HA/DR  effort  in  FY11 

-  Communications  between  air,  ground,  and  ocean-based  assistance  crews  were 
recorded  in  the  All  Partners  Access  Network  (APAN)  collaboration  system 

-  Internet-based  tool  designed  for  information  sharing  during  disaster  relief  efforts 

»  Wikis,  blogs,  forum,  file  sharing,  chat, ... 

-  >  1,700  individuals  used  APAN  during  relief  effort 

-  Multinational  partners,  NGOs,  and  US  Federal  and  State  agencies 

-  Period  covered:  Jan  13,  2010  -  May  26,  2010 

Lexical  link  analysis  was  performed  on  APAN  data 

-  Identified  terms  that  emerged  and  organizations  involved  over  the  course  of  the 
Haiti  HA/DR  operation 

-  MSSQL  database  included:  1173  Forum  posts  and  3900  Blog  messages 

Systematic  way  to  select  a  subset  of  the  vast  amount  of  data  in  APAN  data  set 

-  Provided  a  way  to  isolate  all  communications  related  to  a  particular  topic,  such  as 
'hospital/  'water/  'logistics/  and  'security' 

Applied  definitions  of  macrocognitive  processes  included  in  the  model 

4 


NAVAL 

5i' '  POSTGRADUATE 
SCHOOL 


Technical  Approach 


Lexical  Link  Analysis 

•  Analysis  of  structured  and  unstructured  text  files  (Gallup  &  wood,  2010) 

-  Automated  parsing  of  documents 

-  Develops  themes  for  examined  database  based  on  frequency  of  occurrence  of  key  words 

•  Lexical  analysis  -  text  mining,  word  pairs  extracted  from  documents,  e.g., 
Blog  and  Forum  (zhao,2oio) 


•  Link  analysis  -  reveals  relationships  between  two  objects 

-  Discovers  and  displays  a  network  of  word  pairs 


•  Generated  40  themes  using  LLA 
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•  Collaboration  occurs 


Collaboration 


-  "When  a  group  of  autonomous  stakeholders  of  a  problem  domain  engage  in 
an  interactive  process,  using  shared  rules,  norms,  and  structures,  to  act  or 
decide  on  issues  related  to  that  domain"  (Wood  &  Gray,  1991,  p.11) 


•  Interactive  process  performed  in  a  collaborative  team  environment 

-  Collaborative  defined  as  the  "cognitive  aspects  of  joint  analysis  or  problem 
solving  for  the  purpose  of  attaining  shared  understanding  sufficient  to  achieve 

situational  awareness  for  decision  making  or  creation  of  a  product"  (Letsky  & 
Warner,  2008,  p.  4). 

•  Benefits  afforded  by  inter-organizational  collaboration 

-  Better  decision-making  as  a  result  of  shared  information 

-  Enhanced  coordination  among  dispersed  units 

-  Innovation  resulting  from  the  cross-pollination  of  ideas 

-  Cost  savings  produced  by  sharing  resources  and  the  transfer  of  smart 
practices  (Hansen  &  Nobia,  2004;  Mankin  &  Fitzgerald,  2004). 
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Selecting  data 

•  Large  set  of  real-world  data  with  all  the  characteristics  of  interest  for 
conducting  an  empirical  evaluation  of  the  model  of  team  collaboration 

-  Ad  hoc  teams  are  quickly  assembled  to  deal  with  an  emerging  event  that 
requires  a  collaborative  effort  to  deal  with  the  problem-solving  event 

•  Data  representing  four  themes  was  selected  based  on  LLA 

-  Water,  hospital,  security,  logistics 

Data  analysis  and  coding 

•  One  pair  of  coders  coded  two  themes:  Water,  hospital, 

•  Second  pair  of  coders  coded  two  themes:  Security,  logistics 

•  Cohen's  kappa  used  to  calculate  percentage  agreement 

-  72%  and  70%  agreement  for  water  and  hospital  themes 

Organizing  data 

•  Data  unitized  by  separating  each  thought  unit  on  separate  line 

-  A  thought  unit  refers  to  a  "sequence  of  a  few  words  conveying  a  single 

thought"  (Welden,  Jehn,  &  Pradhan,  1991,  p.  559) 

-  "the  smallest  message  unit  th^TOaW^tSnd  alone"  (Keyton  &  Beck,  2010,  p.  336) 


Practice  coding 

•  Coders  independently  coded  200  lines  from  separate  data  set 

•  Raters  discussed  respective  coding  w/  researcher  to  calibrate  their  use  of  the 
macrocognitive  process  categories 

•  Then  completed  additional  practice  coding  200  lines  of  APAN  data  on  a  separate 


theme 


Independently  coded  data  from  forum  and  blogs  for  their  two  themes 

•  Reviewed  respective  coding,  calculated  percent  agreement,  resolved  any 
differences  in  coding 

Motivated  to  unpack  the  macrocognitive  category  that  contained  the 
majority  of  team  communications.  Team  Information  Exchange  (TIE) 

•  81%  of  water  and  hospital  theme  thought  units  coded  as  TIE 

•  Research  team  reviewed  the  NDM  literature  related  to  sensemaking  to 
determine  additional  cognitive  processes  that  might  be  used  to 


iA 

mill 


differentiate  the  large  number  of  communications  that  were  coded  as  TIE 
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Measurement  Model  of  Team  Collaboration 

(From  Fiore,  Rosen,  Smith-Jentsch,  Salas,  Letsky,  &  Warner,  2010) 


1  IMMl 


Legend: 
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•  A  consistent  pattern  is  evident  across  task  domains  previously  investigated  where 
the  majority  of  thought  units  were  coded  as  representing  two  of  the  cognitive 
processes:  Team  Information  Exchange  (TIE)  and  Team  Knowledge  Sharing  (TKS) 

•  Team  Info  Exchange:  passing  relevant  information  to  the  appropriate  team 
member 

•  Team  Knowledge  Sharing:  explanations  and  interpretations  shared  between  TMs 


Percentage  of  Communications  Coded  as  Team  Information  Exchange  and 
Team  Knowledge  Sharing  across  Task  Domains 


Task  Domain 

Team  Information 
Exchange 

Team  Knowledge 

Sharing 

Firefighters  9-11 

42.01 

27.02 

Maritime  Interdiction  Operation 

60.01 

5.02 

Air  Operations  Center 

5 1 .5 1 

10.32 

NORAD 

52.91 

3.72 

UAV  Dynamic  Re -planning 

58.00 

4.29 

Haiti  HA/DR  Effort 

81.00 

5.90 

Note:  'In  a  previous  version  of  the  model  this  macrocognitive  process  was  labeled  Task  Knowledge  Development. 

2In  a  previous  version  of  the  model  this  macrocognitive  process  was  labeled  Team  Knowledge  Development. 

WWW.NPS.EDU 


•  Focus  on  tasks  performed  in  “the  wild”  -  culled  from  NDM  literature  related  to 
sensemaking  to  develop  additional  list  of  macrocognitive  processes  that  could  be  used 
for  coding  the  present  set  of  data 

•  Sensemaking:  An  essential  cognitive  function  performed  by  a  variety  of  domain 
practitioners  across  a  wide  range  of  real-world  tasks  (Klein,  Phillips,  Rail,  and  Peluso,  2007) 

-  Begins  when  person  becomes  aware  of  a  weakness  in  their  current  comprehension 
of  a  situation,  often  experienced  as  a  surprise,  in  response  to  unexpected  changes 
or  as  a  failure  of  expectations 

-  Sensemaking  is  a  critical  process  for  teams  engaged  in  real-world  domains  where 
practitioners  deal  with  complex,  dynamic,  evolving  situations  that  are  “rich  with 
various  meanings”  (Klein,  et  al,  2007,  p.  1 14) 

•  Data  practitioners  use  to  develop  an  understanding  of  the  situation  are  often  highly 
ambiguous  and  very  complex  and  the  dynamic  events  require  the  decision  maker  to 
dynamically  update  their  understanding  as  the  situation  evolves  over  time 

-  The  frame  that  is  adopted  by  the  practitioner  will  affect  what  data  are  attended  to 
and  how  these  data  items  are  interpreted 

•  When  the  practitioner  notices  data  that  do  not  fit  the  current  frame,  the 


sensemaking  cycle  of  continuously  moving  toward  better  explanations  is 
activated.  Sensemaking  incorporates  consideration  of  the  following  criteria: 
plausibility,  pragmatics,  coherqp^N(ap^,  reasonableness 
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Macrocognitive  Processes  from  the  NDM  Sensemaking  Literature 


Macrocognitive  Process 

Definition 

Anticipatory  Thinking 

(Klein,  Snowden,  &  Lock  Pin,  2007) 
(Weick  &  Sutcliffe,  2001) 

A  critical  macrocognitive  function  of  both  individuals  and  teams.  A 
form  of  sensemaking  that  is  future  oriented  such  as  forming 
expectancies  about  future  events.  Active  attention  management  where 
the  operator  focuses  attention  on  likely  sources  of  critical  information  is 
a  key  characteristic 

Assessing  Risk 

Evaluation  of  the  potential  consequences  for  risk  or  a  danger  to  reach  a 
desired  end-state 

Problem  Detection 

A  form  of  sensemaking  that  recognizes  issues  arising  from  the  current 
situation  where  the  outcome  could  be  detrimental  if  not  addressed 

Planning  and 
Re-planning 

A  process  where  team  members  build  a  list  of  actions  that  will  be 
performed  to  solve  a  problem  and  adjust  as  developments  occur 

Sensemaking 

The  process  of  framing  and  reframing  current  inputs  to  the  problem  in  a 
continuous  process  that  helps  filter  and  interpret  the  data 

Using  Analogues 

Comparing  the  current  situation  with  past  experiences  to  solve  the 
current  situation 

12 
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Results 


•  Previous  research  indicates  non-exhaustive  set  of  macrocognitive  processes 

•  Decision  making  -  what  we  label  Decision  to  Take  Action  (DTA)  -  emerged  during 
previous  analyses  of  six  task  domains  as  a  new  macrocognitive  process 

-  Decision  making  is  an  essential  macrocognitive  process  when  teams  are  involved  in 
responding  to  many  complex,  real-world  tasks 


mail 


Dynamic  decision-making  tasks  require  decisions  throughout  the  entire 
scenario  (Brehmer,  1992) 

-  Deciding  to  take  action  is  viewed  as  both  a  macrocognitive  process  and  a  product 
of  team  collaboration  (Klein,  1993) 

-  Decision  maker  continuously  engaged  in  monitoring  environment,  reassessing  the 
situation,  and  trying  to  understand  what  is  unfolding  until  a  decision  is  called  for 

•  This  view  sees  knowing  when  to  act  as  critical  as  knowing  what  to  do  (Warwick  & 
Hutton,  2007) 

Many  task  domains  require  an  interleaving  of  knowledge  building,  decision 
making  and  taking  action  to  accomplish  the  mission 

-  Opposed  to  the  team  making  one  final  decision  at  the  conclusion  of  the  scenario 

Analysis  of  range  of  task  domains  indicates  types  of  tasks  described  by 
model  typically  involve  team  members  making  decisions  as  part  of  the  1 3 
team’s  collaborative  problem  solvmg*NI  S 
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Decision  Making  is  Part  of  Problem  Solving 
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•  A  decision  can  be  defined  as  a  “mental  event  that  occurs  at  a  singular  point  in 
time... that  leads  immediately  or  directly  to  action”  (Hoffman  &  Yates,  2005,  p.  77). 

-  From  this  perspective,  a  decision  is  a  commitment  to  a  course  of  action. 


•  Decision  making  is  an  essential  component  of  team  collaboration  for  effective 
team  problem  solving  when  team  is  performing  the  task  as  opposed  to 
planning 

•  Consistent  finding  across  seven  task  environments  analyzed: 


-  Firefighters  on  Sept  1 1  responding  to  the  attack  on  the  world  trade  center 

-  Air  warfare  teams  on  a  Navy  ship 

-  Boarding  team  conducting  a  maritime  interdiction  operation 

-  NORAD  collaborating  with  the  FAA  on  Sept  1 1  to  ground  all  commercial  air  traffic 

-  Air  Force  team  responding  during  a  time-sensitive  targeting  scenario 

-  Experiment  involving  UAV  real-time  planning  and  execution 


FDNY 

Air 

Warf 

MIO 

AOC 

NORAD 

Course  of  Action  (COA) 

12 

22 

2 

4.7 

1.2 

Request  Take  Action  (RTA) 

7 

5 

9 

2.8 

4.1 

Total  Percent 

19 

27 

11 

7.5 

5.3 

7.4 


7.4 
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Results 
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Examined  a  wide  range  of  task  domains: 

•  Firefighters  on  9/1 1 

•  Air  warfare  decision  making  on  US  Navy  ship 

•  Maritime  Interdiction  Operations  (MIO) 

•  NORAD/F AA  on  Sept  1 1 ,  200 1 

•  Air  Operations  Center  Dynamic  Targeting  Exercise 

•  UAV  planning  and  execution 

•  Haiti  HA/DR  Operation 

•  Security  -722  thought  units;  Relief  Logistics  1,260 

•  Water  &  Hospital  -  1 ,220  thought  units 

High  inter-rater  reliability: 

•  Firefighters,  9/11  -  89.3% 

•  Air  Ops  Center  dynamic  Targeting  -  89% 

•  FAA/NORAD  -  77% 

•  Haiti  HA/DR  -  72%  and  70%, Water  &  hospital  themes 

Results  indicate  task  environment  will  influence 
which  processes  are  used 

-  C2  teams  collaborate  on  situation  assessment 
but  often  use  RPD  decision  strategy  to 
determine  COA 


Moderation 
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Apply 

Processes 

/ 

< 

model  to 

real- 

>alpul(«)  j 

TNT  -  MIO  Scenario 
Multi-Level  Committee  (of  Teams) 
Decision  Support  (DS)  Structure 

Multiple  DMs  with  complete  participant  interaction  at  multiple 

levels  in  the  chainof  command 
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A  more  complete  model  of  team 
collaboration  can  guide  designers  of 
collaboration  tools  to  facilitate  decision 
making  as  part  of  the  overall  task. 
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Results  of  analysis  of  APAN  data  for  Water  and 

Hospital  Themes 


Macrocognitive  Process  Categories 


Percentage  of  Thought  Units 


Individual  Knowledge  Building 

Water 

Hospital 

IIG 

Individual  Information  Gathering 

4.10 

3.60 

IIS 

Individual  Information  Synthesis 

0.00 

0.00 

KOB 

Knowledge  Object  Development 

0.00 

0.00 

Team  Knowledge  Building 


TIE 

Team  Information  Exchange 

81.00 

81.80 

TKS 

Team  Knowledge  Sharing 

5.90 

5.60 

TSOG 

Team  Solution  Option  Generation 

1.80 

1.50 

TENA 

Team  Evaluation  and  Negotiation  of  Alternatives 

0.00 

0.00 

Decision  to  Take  Action 

COA 

DTA  (Issue  Course  of  Action) 

0.00 

0.00 

RTA 

DTA  (Request  Take  Action) 

7.30 

7.40 

TIE  -  passing  relevant  info  to  appropriate  TMs  at  tti®\apprQpriate  time 


Accomplishments 


Examples  from  the  ‘Hospital’  theme  coded  as  Anticipatory  Thinking. 


Thought  Unit 

Model 

Code 

NDM 

Code 

1  We  have  capacity  for  10  patients  immediately  and  probably 
100  more  in  the  coming  days. 

TIE 

AT 

2.  At  this  point,  like  I  was  saying,  we  have  perhaps  twelve 
hours  of  working  materials  to  keep  going. 

TIE 

AT 

3.  The  use  of  CAP  (combat  air  patrol)  will  avoid  the  logjam  at 

PAP  [Port-au-Prince]  and  is  an  effective  innovation  in  putting 
the  supplies  and  aid  where  they  are  needed. 

TIE 

AT 
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Problem  Detection:  a  form  of  sensemaking  that  recognizes 
an  issue  arising  from  the  current  situation  where  the  outcome 


1. 

2. 

3. 

4. 

5. 

6. 

7. 

8. 
9. 

10. 

11. 

12. 

13. 

14. 


Thought  Unit 

Improvement  in  delivery  method  will  aid  some  and  can  be  implemented  immediately  but  the  final 
solution  will  require  security  on  the  ground. 

Unfortunately  the  situation  is  critical  and  although  US  choppers  are  flying  overhead  regularly  there  is 

as  yet  no  contact  between  one  of  the  few  remaining,  standing  hospitals  in  the  country  and  the  US 
military. 

We  have  run  out  of  antibiotics  and  analgesics. 

500  children  40  miles  N/NE  of  Port-au-Prince  that  have  about  24  hours  of  food  and  water  left!!! 

The  rumor  is  that  force  protection  -  a  Force  protection  requirement  -  is  impeding  aid  delivery. 
Again,  we  have  over  a  thousand  patients  that  are  ready  for  surgery. 

The  hospital  administrator  for  Bernard  Mevs  Hospital  located  near  the  Port-Au-Prince  airport  has 
reported  an  escalating  number  of  pediatric  malaria  cases  requiring  treatment. 

The  hospital  official  noted  a  limited  supply  of  anti-malarial  pharmaceuticals  available. 

From  OCHA  [ Office  for  the  Coordination  of  Humanitarian  Affairs]'.  The  Haiti  emergency  is  also  a 
high-risk  environment  for  sexual  exploitation  and  abuse. 

At  this  point,  like  I  was  saying,  we  have  perhaps  twelve  hours  of  working  materials  to  keep  going. 

In  these  operating  rooms,  we  don’t  have  oxygen,  we  don’t  have  general  anesthesia, 
we  don’t  have  narcotic  pain  medicines  in  enough  quantity. 

Again,  we  have  over  a  thousand  patients  that  are  ready  for  surgery. 

“WEBSITE”  does  not  seem  to  have  contacts  for  hospitals  in  Haiti  and  seems  to  provide  maps  only  in 
PDF  format. 

The  problem  we  are  facing  is  with  discharge. 


Model  NDM 
Code  Code 


TIE 

PD 

TIE 

PD 

TIE 

PD 

TIE 

PD 

TIE 

PD 

TIE 

PD 

TIE 

PD 

TIE 

PD 

TIE 

PD 

TIE 

PD 

TIE 

PD 

TIE 

TIE 

PD 

PD 
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Examples  from  APAN  data  Water  Theme  Coded  as  Problem  Detection 

Thought  Unit  Model  NDM 

Code  Code 

The  U.S.  has  supply  agreements  with  Luxembourg  (attached),  though  it  appears  TIE  PD 

1  they  exclude  North  America. 

2.  At  this  moment  my  condition  is  very  difficult  with  the  orphans  because  we  can’t  TIE  PD 

have  food,  water,  and  medicines  and  food  supplies  because  we  don’t  have 

money  on  hand  to  do  that. 

3.  But  the  roads  are  not  yet  all  named  in  the  Montagne  Noire  area.  TIE  PD 

4-  We  safely  dropped  shelter  boxes  (emergency  tent/filtration/blanket  survival  kits)  TKS  PD 

Friday  but  gents,  this  is  food  and  I  fear  for  their  safety. 

5-  Last  contact  with  the  orphanage  revealed  little  to  no  food,  no  access  to  water  and  TIE  PD 

a  deep  concern  about  possible  criminal  activity  in  the  area. 

6.  Jumping  the  water  from  US  to  Haiti  is  the  only  missing  piece  of  our  puzzle!  TIE  PD 

V.  They  didn’t  have  radios,  batteries  or  generators  and  the  earthquake  silenced  TIE  PD 

landlines  and  mobile  services,  leaving  Haitians  in  the  dark  in  more  ways  than 
one. 

8.  MSC  or  airlift  needed  ASAP.  TIE  PD 

19 
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Results 


•  Analysis  of  a  range  of  task  domains  indicates  that  several  macrocog  processes  cannot 


be  measured  during  certain  types  of  real-world  problem  solving 


•  Two  explanations: 

-  Many  tasks  require  dynamic  decision  making  (Montgomery,  1993;  1989) 

•  Rapid  responses  are  required  to  deal  with  the  event 

•  When  TMs  use  RPD  strategy,  usually  the  situation  itself  either  determines  or 
constrains  the  response  options  and  experienced  decision  makers  make  up  to 
90%  of  all  decisions  w/o  considering  alternatives  (Klein,  1989) 

•  If  the  situation  appears  similar  to  one  that  the  decision  maker  has  previously 
experienced,  the  pattern  will  be  recognized  and  the  course  of  action  is  usually 
immediately  obvious. 

•  Firefighters,  as  well  as  TMs  in  many  other  domains  collaborate  on  the  ‘front 
end’  of  the  problem,  but  do  not  collaborate  on  how  to  respond  to  an  event  due 
to  time  pressure. 

Little  evidence  for  team  evaluation  and  negotiation  of  alternatives  by  team  members 
during  dynamic  decision-making  problem-solving  task  domains  because  many  responses 
are  guided  by  standard  operating  procedures,  and  the  stored  schemas  of  highly 
experienced  operators. 
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•  Many  cognitive  processes  included  in  the  model  are  not  amenable  to 
measurement  when  working  with  real-world  teams 

-  Analyze  chat  logs  or  transcripts  from  large-scale  real-world  events  and  exercises 

•  Typically  not  possible  to  administer  surveys  or  collect  data  to  measure  many  processes  in 
model 

-  See  little  evidence  for  many  processes  included  in  the  model,  such  as  Knowledge  Object 
Development 

•  Some  macrocognitive  processes  included  in  the  Internalized  Team  Knowledge 
Products  Stage  (stage  III)  require  direct  interaction  with  team  members  to  gather  data 
required  to  measure  certain  processes. 

-  Teammate  Knowledge  Similarity ,  Shared  Situation  Awareness ,  and  Task 
Knowledge  Stock ,  are  amenable  to  measurement  in  lab  settings  but  it  is  typically 
not  feasible  to  obtain  measurements  during  a  or  during  a  real-world  event 

•  Team  Problem  Solving  Outcomes  (stage  V)  requires  performance  assessment  metrics  to  assess 
the  Quality  of  the  Plan/Problem  Solving  Solution ,  Efficiency  of  Planning  Process ,  and  Efficiency 
of  Plan  Execution 

•  Performance  metrics  for  the  types  of  task  domains  we  have  studied  do  not  exist 

-  Developing  measurement  strategies  to  assess  how  the  team  performed  overall,  would  require 

a  major  effort  in  itself  21 


WWW.NPS.EDU 


NAVAL 

POSTGRADUATE 

SCHOOL 


Conclusions 


•  Content  analysis  was  used  to  investigate  inter-organizational  collaboration 

•  Added  new  processes  to  coding  scheme  in  an  effort  to  discover  other 
macrocognitive  processes  that  might  be  employed  by  collaborating  teams 

-  Specifically  we  were  interested  in  cognitive  processes  that  are  currently  coded  as  Team 
Information  Exchange 

•  Exploratory  effort  to  discover  whether  additional  cognitive  processes  are 
employed  during  complex,  information-rich,  problem-solving  events 

-  Evidence  for  these  additional  processes,  that  is,  sensemaking,  anticipatory  thinking,  problem 
detection,  assessing  risk,  planning  and  re-planning,  and  using  analogues 

•  Developing  a  more  comprehensive  understanding  of  the  macrocognitive 
processes  involved  in  team  collaboration  has  several  practical  advantages 

-  Conceptualizations  that  provide  the  theoretical  foundation  for  a  model  of  team  collaboration 
that  take  into  account  consistent  findings  based  on  empirical  research  in  real-world  work 
domains  are  likely  to  be  more  accurate 

-  The  way  team  cognitive  processes  influence  team  functioning  was  listed  among  the  top  ten 
critical  research  questions  in  team  research  (Salas  &  Wildman,  2009) 

-  A  more  complete  model  of  team  collaboration  can  guide  designers  of  collaboration  tools  to 

facilitate  decision  making  as  part  of  the  overall  task  22 
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